[IMP] Stop indexing module prose: drop .rst, .md and .css - #2
Open
moylop260 wants to merge 1 commit into
Open
Conversation
Every extension should earn its place by how much of the graph it builds. Measured
on a real Odoo instance (188 modules, 77935 DEFINES edges):
.py 2755 files 55603 definitions 20.2 per file 71.3% of the graph
.xml 1883 files 14844 definitions 7.9 per file 19.0%
.js 890 files 6368 definitions 7.2 per file 8.2%
.scss 232 files 912 definitions 3.9 per file 1.2%
.csv 86 files 86 definitions 1.0 per file 0.1%
.sql 20 files 35 definitions 1.8 per file 0.0%
.rst 73 files 73 definitions 1.0 per file 0.1% <- dropped
.css 8 files 8 definitions 1.0 per file 0.0% <- dropped
.md 2 files 6 definitions 3.0 per file 0.0% <- dropped
".rst" and ".css" build exactly one node each -- the File node itself and nothing
inside it. ".md" only adds the headings of a README: every Section node in that
graph came from its two markdown files. All three are module prose, so the graph
gains a path and no structure.
".csv" scores the same 1.0 and stays anyway. An indexed file is also greppable
through search_code, and ir.model.access.csv is a file people really do search:
searching the indexed graph for "access_" returns 52 hits inside csv files, with
line numbers. Structure is not the only thing an indexed file is good for.
This is about keeping the graph free of prose, not about making a scope fit: the
three dropped extensions are 1.4% of the files of an instance, where .js alone is
15%. Anyone who wants them back has --extensions.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2 +/- ##
==========================================
+ Coverage 92.73% 92.86% +0.12%
==========================================
Files 6 6
Lines 785 799 +14
==========================================
+ Hits 728 742 +14
Misses 57 57 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
SOURCE_GLOBShad never been justified against the graph it produces. Measured on a realOdoo 19 instance (188 modules, 5016 files, 77935
DEFINESedges), counting how manydefinitions each extension contributes:
.py.xml.js.scss.sql.csv.rst.css.md.rstand.cssbuild exactly one node each — the File node itself, and nothing insideit.
.mdonly adds the headings of a README: everySectionnode in that graph (4 of them)came from its two markdown files. All three are module prose, so the graph gains a path and
no structure.
What
SOURCE_GLOBSbecomes.py .xml .js .scss .csv .sql. The reasoning and the measurementlive next to the constant, so the next person changing it argues with numbers.
Why
.csvstays even though it scores the same 1.0Because structure is not the only thing an indexed file is good for:
search_codegreps thecontents of indexed files. Searching the graph above for
access_restricted to.csvreturns 52 hits with line numbers, e.g.
ir.model.access.csvis a file people really do search, so dropping it would cost a realcapability to save nothing.
.rst/.md/.cssare greppable too, but module prose is notwhat anyone greps this graph for.
Not a memory change
Worth being explicit, because it is the obvious thing to assume: this does not make a
large scope fit. The three dropped extensions are 1.4% of the files of an instance, where
.jsalone is 15%. Indexing peak tracks extracted nodes, and those come overwhelmingly from.py. This is about keeping the graph free of prose.Verified
pre-commit run --all-filesclean..csvexception, and that*.scsssurvives — itends with the same three letters as
*.css, so a naive removal would take it too.query_graphover a graph indexed inside theirc190_01container, not from an estimate.